Presentation: Tweet"Applications in the Emerging World of Stream Processing"

Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of continuously changing data in real time? The answer is stream processing along with enabling the collection of large amounts of data in real-time. The most obvious advantage of stream processing is the ability to move many analytical or reporting processes into real time. However, the excitement around stream processing goes well beyond just faster analytics or reporting. What stream processing really enables is the ability to build a company’s business logic and applications around the data that was previously only available in batch fashion in the data warehouse; as well as do that in a continuous fashion rather than once-a-day.

This presentation will give a brief introduction to Apache Kafka and describe it's usage as a platform for streaming data. It will explain how Kafka serves as a foundation for both streaming data pipelines and applications that consume and process real-time data streams. It will introduce some of the newer components of Kafka that help make this possible, including Kafka Connect, framework for capturing continuous data streams, and Kafka Streams, a lightweight stream processing library for use in microservices. Finally it will describe the lessons learned by companies like LinkedIn building massive streaming data architectures.